<html>
<head><meta charset="utf-8"><title>Bug on highly parallel builds · t-compiler/wg-parallel-rustc · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/index.html">t-compiler/wg-parallel-rustc</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html">Bug on highly parallel builds</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="183781873"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183781873" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183781873">(Dec 18 2019 at 19:26)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> What's the nature of the bug with highly parallel builds?</p>



<a name="183782726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183782726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183782726">(Dec 18 2019 at 19:34)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> I think Alex was referring to the fact that at least on Linux we've measured fairly high contention inside the kernel due to our current strategy for limiting parallelism (i.e., GNU make jobserver, which on linux is a machine-global pipe), which can lead to abysmal performance with lots of threads</p>



<a name="183782793"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183782793" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183782793">(Dec 18 2019 at 19:35)</a>:</h4>
<p>we have a few ideas for mitigation and are currently working on the likely solution which is making cargo be the source of truth for jobserver communication (though rustc will support, and always default to, normal jobserver operation)</p>



<a name="183784232"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183784232" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183784232">(Dec 18 2019 at 19:51)</a>:</h4>
<p>yes this has to do with how we're limiting parallelism</p>



<a name="183784282"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183784282" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183784282">(Dec 18 2019 at 19:52)</a>:</h4>
<p>the bug we've seen is that when you <code>write</code> to a pipe on linux, it wakes up <em>everyone</em> waiting on it</p>



<a name="183784310"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183784310" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183784310">(Dec 18 2019 at 19:52)</a>:</h4>
<p>so, in the worst case, when you do <code>-Zthreds=72</code> you have 72^2 threads</p>



<a name="183784316"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183784316" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183784316">(Dec 18 2019 at 19:52)</a>:</h4>
<p>72 rustc's, each with 72 threads</p>



<a name="183784323"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183784323" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183784323">(Dec 18 2019 at 19:52)</a>:</h4>
<p>that's a lot of people to wake up all the time</p>



<a name="183784327"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183784327" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183784327">(Dec 18 2019 at 19:52)</a>:</h4>
<p>and a lot of wasted time churning around</p>



<a name="183784346"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183784346" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183784346">(Dec 18 2019 at 19:52)</a>:</h4>
<p>we plan to fix this by not actually having 72^2 threads waiting</p>



<a name="183784351"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183784351" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183784351">(Dec 18 2019 at 19:52)</a>:</h4>
<p>but only 72 :)</p>



<a name="183785285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785285">(Dec 18 2019 at 20:03)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> This is actually a known bug that's being fixed in Linux <em>right now</em>.</p>



<a name="183785312"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785312" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785312">(Dec 18 2019 at 20:03)</a>:</h4>
<p>There was a large pipe rework in upstream Linux, and in the course of evaluating that, Linus noticed that both the old and the new pipe behavior had the "thundering herd" problem.</p>



<a name="183785333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785333">(Dec 18 2019 at 20:03)</a>:</h4>
<p>So that's getting fixed now, which should substantially speed up <code>make -j72</code> as well.</p>



<a name="183785411"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785411" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785411">(Dec 18 2019 at 20:04)</a>:</h4>
<p>The only reason it isn't <em>already</em> fixed is that there's a bug in <code>make</code> that fixing it uncovered. :)</p>



<a name="183785424"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785424" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785424">(Dec 18 2019 at 20:04)</a>:</h4>
<p>sounds promising!</p>
<p>Do we need to do anything on our side to get that behavior?</p>
<p>I expect we'll still want our changes since macOS etc probably won't get that fix for a while</p>



<a name="183785470"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785470" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785470">(Dec 18 2019 at 20:05)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> You wouldn't need to do anything. The main thing would be that you could just drop the limit to 4 threads on recent Linux.</p>



<a name="183785483"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785483" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785483">(Dec 18 2019 at 20:05)</a>:</h4>
<p>where recent is "unreleased"?</p>



<a name="183785489"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785489" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785489">(Dec 18 2019 at 20:05)</a>:</h4>
<p>Also, does macOS actually have the thundering herd problem?</p>



<a name="183785503"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785503" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785503">(Dec 18 2019 at 20:05)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> Where recent right now is "change not yet in git master, being evaluated".</p>



<a name="183785590"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785590" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785590">(Dec 18 2019 at 20:06)</a>:</h4>
<p>We've not done benchmarking on macOS, but sort of assumed so -- it's also hard to expose as macOS with more than a 4-6 cores is pretty hard to get (modulo just released mac pros)</p>



<a name="183785625"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785625" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785625">(Dec 18 2019 at 20:06)</a>:</h4>
<p>Yeah, that's fair. :)</p>



<a name="183785637"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785637" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785637">(Dec 18 2019 at 20:07)</a>:</h4>
<p>/me really wants someone to build a scalable macOS cloud.</p>



<a name="183785824"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785824" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785824">(Dec 18 2019 at 20:09)</a>:</h4>
<p>I imagine we'll not really get any benefits from these changes for at least a year or two, right? That's my impression of approximate timeline on getting kernel changes pushed into end-user's hands</p>



<a name="183785838"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785838" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785838">(Dec 18 2019 at 20:09)</a>:</h4>
<p>i.e., there's no point in waiting or anything like that</p>



<a name="183785921"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785921" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785921">(Dec 18 2019 at 20:10)</a>:</h4>
<p>so we probably want to move ahead with our fixes regardless</p>



<a name="183785925"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785925" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785925">(Dec 18 2019 at 20:10)</a>:</h4>
<p>even if they get even more improved in the eventuality</p>



<a name="183785961"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785961" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785961">(Dec 18 2019 at 20:11)</a>:</h4>
<p>Also <span class="user-mention" data-user-id="116015">@Alex Crichton</span> -- I think you're not quite right that we'd have 72 threads waiting with our proposed fix, since at least I was expecting that we'd loosely have 1 thread per pipe</p>



<a name="183785977"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785977" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785977">(Dec 18 2019 at 20:11)</a>:</h4>
<p>(just 72 threads total but that's not too interesting)</p>



<a name="183785986"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183785986" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183785986">(Dec 18 2019 at 20:11)</a>:</h4>
<p>The changes should get into one of the next two or so kernels, I'd guess.</p>



<a name="183786044"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183786044" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183786044">(Dec 18 2019 at 20:12)</a>:</h4>
<p>It won't take years.</p>



<a name="183786061"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183786061" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183786061">(Dec 18 2019 at 20:12)</a>:</h4>
<p>Also, there's still value in interoperating with the standard jobserver.</p>



<a name="183786123"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183786123" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183786123">(Dec 18 2019 at 20:13)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> whoa that's awesome! (that linux is fixing the root of the issue)</p>



<a name="183786149"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183786149" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183786149">(Dec 18 2019 at 20:13)</a>:</h4>
<p>Yeah. I'm currently trying to grab the WIP patches and test them, to see how that does.</p>



<a name="183786157"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183786157" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183786157">(Dec 18 2019 at 20:13)</a>:</h4>
<p>Well to be clear our proposal is entirely interoperable, rustc won't lose compat, just be more limited with default jobserver</p>



<a name="183786159"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183786159" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183786159">(Dec 18 2019 at 20:13)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> oh true, it's more of a moral equivalent</p>



<a name="183786176"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183786176" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183786176">(Dec 18 2019 at 20:13)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> so far we tested out switching to literal posix semaphores, and that simple change ended up fixing the scaling issues effectively on my 28-core machine</p>



<a name="183786227"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183786227" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183786227">(Dec 18 2019 at 20:14)</a>:</h4>
<p>it's still not <em>perfect</em> because rustc immediately spawns ncores threads, which is quite a lot</p>



<a name="183786233"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183786233" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183786233">(Dec 18 2019 at 20:14)</a>:</h4>
<p>so we need to fix that a bit, but the scaling was <em>much</em> better with posix semaphores</p>



<a name="183786239"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183786239" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183786239">(Dec 18 2019 at 20:14)</a>:</h4>
<p>(where sempahores presumably don't have the thundering herd issue)</p>



<a name="183786255"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183786255" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183786255">(Dec 18 2019 at 20:14)</a>:</h4>
<p>but yeah we're unlikely to stick with vanilla jobservers for macos/windows which are likely to have similar problems</p>



<a name="183787323"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787323" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787323">(Dec 18 2019 at 20:28)</a>:</h4>
<p>Windows has pipe-equivalents that guarantee to only wake up one waiter.</p>



<a name="183787338"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787338" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787338">(Dec 18 2019 at 20:28)</a>:</h4>
<p>I don't know macOS internals well enough to know if it does.</p>



<a name="183787364"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787364" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787364">(Dec 18 2019 at 20:28)</a>:</h4>
<p>I think I'm mostly trying to figure out if you can <em>avoid</em> doing that work by instead making sure you're using the right OS primitive.</p>



<a name="183787406"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787406" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787406">(Dec 18 2019 at 20:29)</a>:</h4>
<p>Hypothetically, if Windows and macOS could work around this problem by using something pipe-like that isn't a POSIX pipe, and Linux worked fine with recent kernels and we could limit to 4 threads on older kernels, would you still want to do the extra architectural work for a workaround?</p>



<a name="183787681"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787681" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787681">(Dec 18 2019 at 20:32)</a>:</h4>
<p>I at least expect that there are other advantages to letting Cargo know this information -- for example, I've had some thoughts about scheduling jobserver tokens down in a more intelligent way than what any system primitive would give us I think</p>



<a name="183787716"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787716" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787716">(Dec 18 2019 at 20:33)</a>:</h4>
<p>How so?</p>



<a name="183787723"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787723" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787723">(Dec 18 2019 at 20:33)</a>:</h4>
<p>/me is interested in ways to improve parallel builds.</p>



<a name="183787804"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787804" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787804">(Dec 18 2019 at 20:34)</a>:</h4>
<p>e.g. right now if you're building lots of crates it might make sense to make sure each rustc has 2 threads (presuming you have the cores) rather than have one rustc with 10 threads and the rest stuck at 1 or something like that</p>



<a name="183787841"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787841" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787841">(Dec 18 2019 at 20:35)</a>:</h4>
<p>I mean, as long as you stay 100% CPU bound, you're doing work that will need to happen regardless. The problem comes in if you drop below 100% CPU.</p>



<a name="183787859"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787859" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787859">(Dec 18 2019 at 20:35)</a>:</h4>
<p>well, yes, but it's not necessarily true that you want to e.g. run codegen immediately for some crate if you won't need it for a while</p>



<a name="183787870"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787870" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787870">(Dec 18 2019 at 20:35)</a>:</h4>
<p>Yeah, scheduling at a crate level may make sense.</p>



<a name="183787912"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787912" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787912">(Dec 18 2019 at 20:36)</a>:</h4>
<p>On that note, though...</p>



<a name="183787930"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787930" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787930">(Dec 18 2019 at 20:36)</a>:</h4>
<p>At that point it might make sense to adopt a "pull" model where we try to build <em>later</em> crates and let those drive what we build next.</p>



<a name="183787949"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787949" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787949">(Dec 18 2019 at 20:36)</a>:</h4>
<p>Simulating the idea of earliest-deadline-first.</p>



<a name="183787992"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787992" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787992">(Dec 18 2019 at 20:37)</a>:</h4>
<p>yes, or something like that -- certainly right now rustc isn't really capable of communicating that information up in an on demand fashion</p>



<a name="183787995"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183787995" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183787995">(Dec 18 2019 at 20:37)</a>:</h4>
<p>For instance, almost any project I build that has syn in its dependency graph tends to wind up at a bottleneck where it's building <em>only</em> syn at some point.</p>



<a name="183788000"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183788000" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183788000">(Dec 18 2019 at 20:37)</a>:</h4>
<p>but e.g. rust analyzer I think has a model like this</p>



<a name="183788008"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183788008" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183788008">(Dec 18 2019 at 20:37)</a>:</h4>
<p>So it'd be nice to start syn as soon as possible.</p>



<a name="183788141"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183788141" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183788141">(Dec 18 2019 at 20:39)</a>:</h4>
<p>I think that's kind of what I'm getting at -- we would potentially want to give syn as many threads as we can</p>



<a name="183788208"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183788208" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183788208">(Dec 18 2019 at 20:40)</a>:</h4>
<p>and no system primitive would allow that level of prioritization I expect</p>



<a name="183788379"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183788379" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183788379">(Dec 18 2019 at 20:42)</a>:</h4>
<p>True.</p>



<a name="183788415"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183788415" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183788415">(Dec 18 2019 at 20:42)</a>:</h4>
<p>I think the ideal model effectively looks like "run everything at once, at two priority levels, higher for pull and lower for things we know we'll need eventually but we don't need yet".</p>



<a name="183788552"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183788552" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183788552">(Dec 18 2019 at 20:44)</a>:</h4>
<p>You know the "readahead" trick on Linux, of "figure out everything you need to read from disk, and next time start reading it in the order you'll need it"? We could get a decent approximation of the same thing if we have an estimate for how long (in CPU time) each crate takes to build, assume that future builds will take similar amounts of time, and prioritize accordingly.</p>



<a name="183788589"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183788589" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183788589">(Dec 18 2019 at 20:45)</a>:</h4>
<p>Anyway, I think I've found the Linux patch I need to test.</p>



<a name="183788973"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183788973" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183788973">(Dec 18 2019 at 20:50)</a>:</h4>
<p>I would agree that we probably want our own scheduling system even if linux/windows are fixed</p>



<a name="183788993"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183788993" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183788993">(Dec 18 2019 at 20:50)</a>:</h4>
<p>I think at the bare minimum I agree that this linux kernel change may take a very long time to propagate, and we want to ramp up default parallelism sooner</p>



<a name="183789012"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183789012" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183789012">(Dec 18 2019 at 20:51)</a>:</h4>
<p>but I also agree that we can probably more cleverly prioritize crates</p>



<a name="183789022"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183789022" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183789022">(Dec 18 2019 at 20:51)</a>:</h4>
<p>e.g. cargo already has some degree of scheduling heuristics</p>



<a name="183789026"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183789026" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183789026">(Dec 18 2019 at 20:51)</a>:</h4>
<p>which can help keep everything saturated in theory</p>



<a name="183789043"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183789043" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183789043">(Dec 18 2019 at 20:51)</a>:</h4>
<p>and we could apply similar heuristics to "we have a token, and N rustc instances want a token, who gets it?"</p>



<a name="183789384"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183789384" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183789384">(Dec 18 2019 at 20:55)</a>:</h4>
<p>And we could always simplify the logic in the future, if we ever get to the point that we can start relying on fixed versions.</p>



<a name="183790011"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183790011" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183790011">(Dec 18 2019 at 21:03)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> oh another thing you may want to do for more parallelism on a 72-core machine is to set <code>-Ccodegen-units=100000000</code></p>



<a name="183790015"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183790015" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183790015">(Dec 18 2019 at 21:03)</a>:</h4>
<p>or just set <code>CARGO_INCREMENTAL=1</code> in release mode</p>



<a name="183790029"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183790029" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183790029">(Dec 18 2019 at 21:03)</a>:</h4>
<p>which may drop perf a bit but not a huge amount due to thinlto</p>



<a name="183791155"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183791155" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183791155">(Dec 18 2019 at 21:15)</a>:</h4>
<p>I'll give those a try too, once I have numbers for baseline and <code>-Zthreads=72</code> with the patch I'm testing.</p>



<a name="183791249"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183791249" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183791249">(Dec 18 2019 at 21:16)</a>:</h4>
<p>man that's a whole new world for me, compiling a new kernel and then running it locally <span aria-label="ghost" class="emoji emoji-1f47b" role="img" title="ghost">:ghost:</span></p>



<a name="183792094"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183792094" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183792094">(Dec 18 2019 at 21:26)</a>:</h4>
<p>I'm also using the compile of the kernel I'm testing to cross-check the results. :)</p>



<a name="183792118"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183792118" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183792118">(Dec 18 2019 at 21:26)</a>:</h4>
<p>Kernel compile, on the kernel without the patch:</p>
<div class="codehilite"><pre><span></span>real    0m41.377s
user    10m1.718s
sys 2m25.655s
</pre></div>



<a name="183793890"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183793890" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183793890">(Dec 18 2019 at 21:48)</a>:</h4>
<p>Kernel compile on the kernel with the patch:</p>
<div class="codehilite"><pre><span></span>real    0m40.463s
user    9m55.603s
sys 2m25.449s
</pre></div>


<p>Looks like it doesn't make <em>that</em> much difference to kernel compiles, but considering that half that time is serialized compressing/linking at the end anyway, that's still decent.</p>



<a name="183793900"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183793900" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183793900">(Dec 18 2019 at 21:48)</a>:</h4>
<p>Now to try wasmtime...</p>



<a name="183794188"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794188" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794188">(Dec 18 2019 at 21:52)</a>:</h4>
<p>Didn't help the wasmtime build with the default threads=4:</p>
<div class="codehilite"><pre><span></span>real    1m10.096s
user    17m38.136s
sys 0m24.415s
</pre></div>



<a name="183794276"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794276" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794276">(Dec 18 2019 at 21:52)</a>:</h4>
<p>that looks pretty suspiciously high</p>



<a name="183794285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794285">(Dec 18 2019 at 21:53)</a>:</h4>
<p>I would expect system time to be far lower</p>



<a name="183794296"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794296" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794296">(Dec 18 2019 at 21:53)</a>:</h4>
<p>maybe that's with cold disk cache?</p>



<a name="183794385"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794385" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794385">(Dec 18 2019 at 21:54)</a>:</h4>
<p>Nope, that's the second build.</p>



<a name="183794418"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794418" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794418">(Dec 18 2019 at 21:54)</a>:</h4>
<p>For reference, the wasmtime build on the unpatched kernel with parallel rustc (and default threads) was:</p>
<div class="codehilite"><pre><span></span>real    1m9.191s
user    16m55.383s
sys 0m23.964s
</pre></div>



<a name="183794437"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794437" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794437">(Dec 18 2019 at 21:54)</a>:</h4>
<p>Which is basically the same wall-clock (slightly better) and noticeably better user time.</p>



<a name="183794724"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794724" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794724">(Dec 18 2019 at 21:58)</a>:</h4>
<p>The kernel patch did slightly improve the results with <code>RUSTFLAGS=-Zthreads=72</code>, but that's still much worse than 4:</p>
<div class="codehilite"><pre><span></span>real    1m27.163s
user    40m44.138s
sys 3m4.341s
</pre></div>



<a name="183794744"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794744" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794744">(Dec 18 2019 at 21:58)</a>:</h4>
<p>hm, I think I was remembering when we were benchmarking first 3 seconds of a build which might explain my expectation of lower system times</p>



<a name="183794777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794777">(Dec 18 2019 at 21:59)</a>:</h4>
<p>could you try on both with <code>timeout 3s cargo ...</code>?</p>



<a name="183794785"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794785" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794785">(Dec 18 2019 at 21:59)</a>:</h4>
<p>For comparison, <code>-Zthreads=72</code> <em>without</em> this kernel patch gave 2m1s real-time and 50m user.</p>



<a name="183794821"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794821" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794821">(Dec 18 2019 at 21:59)</a>:</h4>
<p>So the kernel patch to fix thundering herd wakeups for pipes is a <em>huge</em> relative improvement for -Zthreads=72, but there's still a problem there.</p>



<a name="183794884"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794884" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794884">(Dec 18 2019 at 22:00)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> This system takes a while to reboot, so I'd like to get all the results I can with the kernel patch before I switch back to without.</p>



<a name="183794904"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794904" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794904">(Dec 18 2019 at 22:00)</a>:</h4>
<p>ah, okay, wasn't sure if you were working on a separate machine or something</p>



<a name="183794944"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794944" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794944">(Dec 18 2019 at 22:01)</a>:</h4>
<p>I'm doing these tests on a separate system than the one I'm chatting with, but that separate system takes minutes to reboot.</p>



<a name="183794966"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794966" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794966">(Dec 18 2019 at 22:01)</a>:</h4>
<p>(server BIOSes, sigh)</p>



<a name="183794993"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183794993" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183794993">(Dec 18 2019 at 22:01)</a>:</h4>
<p>fwiw, one thing that might be helpful is to get some loose syscall counts/timing</p>



<a name="183795012"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183795012" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183795012">(Dec 18 2019 at 22:01)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> But purely subjectively, without the kernel patch, cargo spent a few seconds showing 0 crates compiled, while <em>with</em> the patch, the "slow startup" problem doesn't seem to be true anymore.</p>



<a name="183795080"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183795080" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183795080">(Dec 18 2019 at 22:02)</a>:</h4>
<p>that's the primary behavior we're expecting to fix, so that's good to hear</p>



<a name="183795402"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183795402" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183795402">(Dec 18 2019 at 22:06)</a>:</h4>
<p>I can easily get some <code>-Ztimings</code> data from this patched kernel.</p>



<a name="183795420"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183795420" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183795420">(Dec 18 2019 at 22:06)</a>:</h4>
<p>Also, I'm trying some experiments with different <code>-Zthreads</code> values.</p>



<a name="183795444"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183795444" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183795444">(Dec 18 2019 at 22:07)</a>:</h4>
<p>For instance, <code>-Zthreads=16</code> gives:</p>
<div class="codehilite"><pre><span></span>real    1m7.279s
user    18m13.828s
sys 0m38.424s
</pre></div>



<a name="183795464"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183795464" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183795464">(Dec 18 2019 at 22:07)</a>:</h4>
<p>That's a bit more user time, but a few seconds wall-clock improvement over 4 threads, and the improvement definitely isn't noise.</p>



<a name="183795540"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183795540" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183795540">(Dec 18 2019 at 22:08)</a>:</h4>
<p>Subjectively, this feels like there's some critical non-linear scaling issue that dominates with high numbers of threads but not with low numbers of threads.</p>



<a name="183795719"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183795719" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183795719">(Dec 18 2019 at 22:11)</a>:</h4>
<p>I tend to agree -- I think we've not yet tracked down what that is. It might be that we just don't have enough data or so, or maybe we're not yet parallel enough (i.e., the compiler itself is not sufficiently able to utilize that parallelism)</p>



<a name="183796705"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183796705" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183796705">(Dec 18 2019 at 22:25)</a>:</h4>
<p>/me would be happy to help get the data you need.</p>



<a name="183796824"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183796824" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183796824">(Dec 18 2019 at 22:26)</a>:</h4>
<p>Do you have experience or time to dig in? One of the problems we've had is that none of us really know how to figure out what the problem is (or what the tools are really either)</p>



<a name="183796838"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183796838" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183796838">(Dec 18 2019 at 22:26)</a>:</h4>
<p>This is <em>entirely</em> subjective, but having worked on parallel programming for many years, this doesn't feel like "we don't have enough parallelism opportunities in the compiler". This feels like either "something is scaling quadratically or worse" or "something is blocking/spinning/waking-up unnecessarily".</p>



<a name="183796847"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183796847" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183796847">(Dec 18 2019 at 22:27)</a>:</h4>
<p>I do have the experience, and some of the time.</p>



<a name="183796869"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183796869" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183796869">(Dec 18 2019 at 22:27)</a>:</h4>
<p>(Parallel programming was my dissertation topic. ;) )</p>



<a name="183796891"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183796891" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183796891">(Dec 18 2019 at 22:27)</a>:</h4>
<p>We'd be happy to receive help, and if you can even write up some quick "things to look at" or so that would be amazing</p>



<a name="183796948"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183796948" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183796948">(Dec 18 2019 at 22:28)</a>:</h4>
<p>I can try and help out with how to get builds and such with the parallelism enabled</p>



<a name="183797158"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797158" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797158">(Dec 18 2019 at 22:30)</a>:</h4>
<p>It depends on the nature of the problem you're debugging. But to a first approximation, if the problem is "too much user time", perf is great for this.</p>



<a name="183797171"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797171" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797171">(Dec 18 2019 at 22:30)</a>:</h4>
<p>("too much blocking" is harder to debug, but possible.)</p>



<a name="183797197"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797197" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797197">(Dec 18 2019 at 22:31)</a>:</h4>
<p>I think we've not had any major success with basic use of perf -- i.e., parallel compiler does not differ from non-parallel</p>



<a name="183797205"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797205" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797205">(Dec 18 2019 at 22:31)</a>:</h4>
<p>(if you mean just <code>perf record</code> and friends)</p>



<a name="183797274"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797274" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797274">(Dec 18 2019 at 22:32)</a>:</h4>
<p>That'd be really surprising. If user time is going from (in my case) 16 minutes to 40-50 minutes, perf really should point to <em>something</em> there...</p>



<a name="183797286"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797286" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797286">(Dec 18 2019 at 22:32)</a>:</h4>
<p>A thought crosses my mind...</p>



<a name="183797308"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797308" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797308">(Dec 18 2019 at 22:32)</a>:</h4>
<p>It's possible we've just not spent enough time on high enough core machines</p>



<a name="183797318"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797318" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797318">(Dec 18 2019 at 22:32)</a>:</h4>
<p>Sometimes, these scaling issues show up <em>much</em> better on bigger systems, precisely because there's more contention. They also can show up more on multi-socket systems, because unnecessary communication is slower between sockets.</p>



<a name="183797344"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797344" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797344">(Dec 18 2019 at 22:33)</a>:</h4>
<p>O(n^2) gets much more noticeable at 72 than 16. :)</p>



<a name="183797347"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797347" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797347">(Dec 18 2019 at 22:33)</a>:</h4>
<p>(I also don't recall specifically looking at this in recent time, though I do recall doing something like this a few months ago, so things may have also changed since then)</p>



<a name="183797368"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797368" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797368">(Dec 18 2019 at 22:33)</a>:</h4>
<p>So, in addition to me personally working on this: is there someone on the parallel rustc team who has room to host an 88-way box somewhere? I'll mail them one. :)</p>



<a name="183797449"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797449" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797449">(Dec 18 2019 at 22:34)</a>:</h4>
<p>It's rackmount hardware, and produces enough noise that you don't want it under or atop your desk, but if someone has somewhere to rack it and make it available for everyone on the parallel rustc team to use it...</p>



<a name="183797473"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797473" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797473">(Dec 18 2019 at 22:35)</a>:</h4>
<p>I am uncertain. I suspect the answer might be no -- and we might have more luck with just one-off running something on EC2, though I'd need to take a look at pricing there.</p>



<a name="183797501"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797501" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797501">(Dec 18 2019 at 22:35)</a>:</h4>
<p>Pricing there is about $5/hour for a comparable box.</p>



<a name="183797584"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797584" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797584">(Dec 18 2019 at 22:36)</a>:</h4>
<p>That's not counting storage and bandwidth.</p>



<a name="183797612"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797612" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797612">(Dec 18 2019 at 22:37)</a>:</h4>
<p>(We do have some budget from AWS so we might be able to afford it, I'm just not sure off the top of my head :)</p>



<a name="183797783"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797783" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797783">(Dec 18 2019 at 22:40)</a>:</h4>
<p>Just confirmed, <code>c5.metal</code> is what you want, which is $4.08/hour.</p>



<a name="183797834"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797834" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797834">(Dec 18 2019 at 22:40)</a>:</h4>
<p>That's a 96-way parallel system.</p>



<a name="183797908"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183797908" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183797908">(Dec 18 2019 at 22:41)</a>:</h4>
<p>(That said, I would love to give this hardware to Rust, where it'll be free for anyone to do this kind of testing on an ongoing basis.)</p>



<a name="183798001"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183798001" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183798001">(Dec 18 2019 at 22:42)</a>:</h4>
<p>It's one of my desires to find a way for the infra team to be able to accept this sort of hardware donation :)</p>



<a name="183798064"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183798064" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183798064">(Dec 18 2019 at 22:43)</a>:</h4>
<p>It doesn't <em>have</em> to be the infra team. I'd accept "someone in the project can arrange to rack it, one-off". So if anyone has an office near a lab...</p>



<a name="183798083"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183798083" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183798083">(Dec 18 2019 at 22:43)</a>:</h4>
<p>(It can always become infra later when there's a path for that.)</p>



<a name="183798128"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183798128" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183798128">(Dec 18 2019 at 22:44)</a>:</h4>
<p>sure, yeah, I agree with that</p>



<a name="183799092"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799092" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799092">(Dec 18 2019 at 22:58)</a>:</h4>
<p>Are there debug symbols available that would allow directly running perf on the nightly-2019-12-18 build, or am I going to need to build from source?</p>



<a name="183799300"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799300" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799300">(Dec 18 2019 at 23:00)</a>:</h4>
<p>you're going to need to build from source unfortunately</p>



<a name="183799334"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799334" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799334">(Dec 18 2019 at 23:01)</a>:</h4>
<p>you can either directly checkout the SHA of that nightly or toggle <code>parallel-compiler = true</code> in config.toml (not sure how much experience you have with rustc compiler dev)</p>



<a name="183799403"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799403" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799403">(Dec 18 2019 at 23:02)</a>:</h4>
<p>Fairly little. Mind pointing me to the "build rustc and cargo from source and <code>rustup link</code> them 101"? :)</p>



<a name="183799468"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799468" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799468">(Dec 18 2019 at 23:03)</a>:</h4>
<p>Even without source, though, I'm seeing some obvious perf issues:</p>
<div class="codehilite"><pre><span></span>  13.07%  rustc            librustc_driver-0d78d9a30be443c5.so          [.] std::thread::local::LocalKey&lt;T&gt;::try_with
  10.95%  rustc            librustc_driver-0d78d9a30be443c5.so          [.] crossbeam_epoch::internal::Global::try_advance
   6.93%  rustc            [unknown]                                    [k] 0xffffffff91a00163
   5.86%  rustc            librustc_driver-0d78d9a30be443c5.so          [.] crossbeam_deque::Stealer&lt;T&gt;::steal
   4.14%  rustc            librustc_driver-0d78d9a30be443c5.so          [.] &lt;core::iter::adapters::chain::Chain&lt;A,B&gt; as core::iter::traits::iterator::Iterator&gt;::try_fold
</pre></div>



<a name="183799489"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799489" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799489">(Dec 18 2019 at 23:03)</a>:</h4>
<p>I'm going to grab a similar profile <em>without</em> <code>-Zthreads=72</code> and see what things stand out as differences.</p>



<a name="183799579"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799579" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799579">(Dec 18 2019 at 23:05)</a>:</h4>
<p>you shouldn't need to build cargo (at least yet) -- it doesn't change for this</p>



<a name="183799588"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799588" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799588">(Dec 18 2019 at 23:05)</a>:</h4>
<p>Good to know.</p>



<a name="183799591"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799591" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799591">(Dec 18 2019 at 23:05)</a>:</h4>
<p>otherwise the doc to look at is <a href="https://rust-lang.github.io/rustc-guide/building/how-to-build-and-run.html" target="_blank" title="https://rust-lang.github.io/rustc-guide/building/how-to-build-and-run.html">https://rust-lang.github.io/rustc-guide/building/how-to-build-and-run.html</a></p>



<a name="183799676"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799676" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799676">(Dec 18 2019 at 23:06)</a>:</h4>
<p>which essentially boils down to cloning the repo, and then this sequence of commands or so:</p>
<div class="codehilite"><pre><span></span>cp config.toml.example config.toml
$EDITOR config.toml
# edit parallel-compiler to be &quot;true&quot; and uncomment it
./x.py build --stage 1 src/libtest
rustup toolchain link stage1 build/&lt;host-triple&gt;/stage1
# cd elsewhere
cargo +stage1 build ...
</pre></div>



<a name="183799686"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799686" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799686">(Dec 18 2019 at 23:07)</a>:</h4>
<p>and you'll only need to run the link step once</p>



<a name="183799698"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799698" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799698">(Dec 18 2019 at 23:07)</a>:</h4>
<p>ah, you might actually need to enable debug symbols as well</p>



<a name="183799709"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799709" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799709">(Dec 18 2019 at 23:07)</a>:</h4>
<p>as <a href="https://rust-lang.github.io/rustc-guide/building/how-to-build-and-run.html#create-a-configtoml" target="_blank" title="https://rust-lang.github.io/rustc-guide/building/how-to-build-and-run.html#create-a-configtoml">https://rust-lang.github.io/rustc-guide/building/how-to-build-and-run.html#create-a-configtoml</a> notes</p>



<a name="183799710"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799710" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799710">(Dec 18 2019 at 23:07)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> ^</p>



<a name="183799713"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799713" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799713">(Dec 18 2019 at 23:07)</a>:</h4>
<p>Thanks, I'll give that a try.</p>



<a name="183799766"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799766" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799766">(Dec 18 2019 at 23:08)</a>:</h4>
<p>Also, yeah, just a simple <code>perf record</code> is showing some massive scaling issues.</p>



<a name="183799777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799777">(Dec 18 2019 at 23:08)</a>:</h4>
<p>With the default threads=4, a parallel rustc has free and malloc at the top of the profile.</p>



<a name="183799785"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799785" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799785">(Dec 18 2019 at 23:08)</a>:</h4>
<p>And various parts of <code>libLLVM-9-rust-1.41.0-nightly.so</code>.</p>



<a name="183799811"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799811" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799811">(Dec 18 2019 at 23:09)</a>:</h4>
<p>With threads=72, all of the top hits look like scaling failures.</p>



<a name="183799908"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799908" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799908">(Dec 18 2019 at 23:10)</a>:</h4>
<p>aha, yeah, so looks like we probably just weren't benchmarking with enough threds</p>



<a name="183799944"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799944" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799944">(Dec 18 2019 at 23:11)</a>:</h4>
<p>the first thing I'd look at is probably bumping the shard bits here: <a href="https://github.com/rust-lang/rust/blob/master/src/librustc_data_structures/sharded.rs#L13-L17" target="_blank" title="https://github.com/rust-lang/rust/blob/master/src/librustc_data_structures/sharded.rs#L13-L17">https://github.com/rust-lang/rust/blob/master/src/librustc_data_structures/sharded.rs#L13-L17</a></p>



<a name="183799952"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183799952" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183799952">(Dec 18 2019 at 23:11)</a>:</h4>
<p>(should be a matter of bumping that constant up)</p>



<a name="183800095"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800095" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800095">(Dec 18 2019 at 23:13)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> another issue is that there's a branch of rayon which in theory greatly improves threads going to sleep</p>



<a name="183800097"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800097" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800097">(Dec 18 2019 at 23:13)</a>:</h4>
<p>that may help a bit here as well</p>



<a name="183800102"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800102" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800102">(Dec 18 2019 at 23:13)</a>:</h4>
<p>this is great to have this system to test on though</p>



<a name="183800170"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800170" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800170">(Dec 18 2019 at 23:14)</a>:</h4>
<p>At the moment, it <em>looks</em> like the bottleneck may be in crossbeam.</p>



<a name="183800204"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800204" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800204">(Dec 18 2019 at 23:15)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> another possibility perhaps, if you can find one crate that isn't getting nearly the speedup you'd think, I was taking a look at <code>-Z self-profile</code> graphs and saw very little work actually being stolen</p>



<a name="183800217"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800217" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800217">(Dec 18 2019 at 23:15)</a>:</h4>
<p>so it may just be that rustc isn't great at saturating cores right now</p>



<a name="183800233"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800233" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800233">(Dec 18 2019 at 23:15)</a>:</h4>
<p>I'm less worried about that, and more worried about the massively increased user time (total CPU time spent) when increasing the number of threads.</p>



<a name="183800322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800322">(Dec 18 2019 at 23:17)</a>:</h4>
<p>Interestingly, it looks like the pile of CPU spent on TLS in <code>std::thread::local::LocalKey&lt;T&gt;::try_with</code> is coming from <code>crossbeam_deque::Stealer&lt;T&gt;::steal</code>.</p>



<a name="183800355"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800355" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800355">(Dec 18 2019 at 23:17)</a>:</h4>
<p>that sounds like plausibly rayon not quite doing a good job</p>



<a name="183800489"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800489" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800489">(Dec 18 2019 at 23:19)</a>:</h4>
<p>Would you recommend testing the same commit that nightly-2019-12-18 used, or testing latest master and just enabling parallelism?</p>



<a name="183800556"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800556" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800556">(Dec 18 2019 at 23:20)</a>:</h4>
<p>shouldn't matter in practice</p>



<a name="183800562"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800562" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800562">(Dec 18 2019 at 23:20)</a>:</h4>
<p>might be a bit harder to test latest master</p>



<a name="183800570"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800570" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800570">(Dec 18 2019 at 23:20)</a>:</h4>
<p>but if you're setting threads up already then you should be fine</p>



<a name="183800584"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800584" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800584">(Dec 18 2019 at 23:20)</a>:</h4>
<p>(i.e., current master does not default to 4 threads but rather 1)</p>



<a name="183800689"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800689" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800689">(Dec 18 2019 at 23:22)</a>:</h4>
<p>I'm fine with manually specifying <code>RUSTFLAGS=-Zthreads=4</code>. :)</p>



<a name="183800719"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800719" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800719">(Dec 18 2019 at 23:22)</a>:</h4>
<p>What about the pile of other changes that got made and reverted?</p>



<a name="183800737"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800737" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800737">(Dec 18 2019 at 23:22)</a>:</h4>
<p>I saw a bunch of type-related changes, for instance (usize vs u64...).</p>



<a name="183800812"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800812" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800812">(Dec 18 2019 at 23:23)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> What's the rayon branch that improves threads going to sleep?</p>



<a name="183800819"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800819" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800819">(Dec 18 2019 at 23:23)</a>:</h4>
<p>shouldn't matter in practice</p>



<a name="183800885"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800885" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800885">(Dec 18 2019 at 23:24)</a>:</h4>
<p>Alright.</p>



<a name="183800894"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800894" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800894">(Dec 18 2019 at 23:24)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> the rayon branch should help with user time because the current bug for rayon is that threads take way too long to go to sleep, which would burn a lot of user time</p>



<a name="183800925"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800925" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800925">(Dec 18 2019 at 23:24)</a>:</h4>
<p>That makes sense; that's absolutely worth testing.</p>



<a name="183800940"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800940" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800940">(Dec 18 2019 at 23:25)</a>:</h4>
<p>the branch was last tested in <a href="https://github.com/rust-lang/rust/pull/66608" target="_blank" title="https://github.com/rust-lang/rust/pull/66608">https://github.com/rust-lang/rust/pull/66608</a></p>



<a name="183800953"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800953" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800953">(Dec 18 2019 at 23:25)</a>:</h4>
<p>I'm not sure if it's easily switchable-too</p>



<a name="183800958"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183800958" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183800958">(Dec 18 2019 at 23:25)</a>:</h4>
<p>we're mostly waiting on rayon to merge the changes itself :)</p>



<a name="183801040"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183801040" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183801040">(Dec 18 2019 at 23:26)</a>:</h4>
<p>Are the changes solid enough that they're expected to be merged, or do they need further hammering-on?</p>



<a name="183801045"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183801045" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183801045">(Dec 18 2019 at 23:26)</a>:</h4>
<p>It doesn't look too hard to switch to.</p>



<a name="183801217"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183801217" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183801217">(Dec 18 2019 at 23:29)</a>:</h4>
<p>I'd expect some of <a href="https://tokio.rs/blog/2019-10-scheduler/" target="_blank" title="https://tokio.rs/blog/2019-10-scheduler/">https://tokio.rs/blog/2019-10-scheduler/</a> could apply well to Rayon too. I don't think anyone has optimized Rayon for 72 threads.</p>



<a name="183801438"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183801438" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183801438">(Dec 18 2019 at 23:32)</a>:</h4>
<p>That does look interesting, thank you for the pointer.</p>



<a name="183801607"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183801607" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183801607">(Dec 18 2019 at 23:35)</a>:</h4>
<p>Having a backoff from stealing stuff when idling would probably help a bit too. Rayon just tries to steal stuff in a loop currently</p>



<a name="183801672"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183801672" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183801672">(Dec 18 2019 at 23:36)</a>:</h4>
<p>Which makes sense if you expect to saturate the system, but not if you expect to cooperate with other parallel work.</p>



<a name="183801720"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183801720" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183801720">(Dec 18 2019 at 23:37)</a>:</h4>
<p>Successfully built rustc master (<code>Build completed successfully in 0:12:13</code>, most of which was LLVM), trying that out to see if I get comparable results.</p>



<a name="183801835"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183801835" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183801835">(Dec 18 2019 at 23:39)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> I don't know the status of the rayon changes myself, <span class="user-mention" data-user-id="116009">@nikomatsakis</span> would be able to speak more to that (the status of the rayon branch and how close it is to landing upstream)</p>



<a name="183803267"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183803267" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183803267">(Dec 19 2019 at 00:01)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> I tried building a parallel rustc based on your instructions, but even with <code>RUSTFLAGS=-Zthreads=72</code> (or 4) I don't seem to get any parallelism.</p>



<a name="183803276"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183803276" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183803276">(Dec 19 2019 at 00:01)</a>:</h4>
<p>/me will try again later.</p>



<a name="183803466"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183803466" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183803466">(Dec 19 2019 at 00:02)</a>:</h4>
<p>hm okay</p>



<a name="183803498"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183803498" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183803498">(Dec 19 2019 at 00:02)</a>:</h4>
<p>you might need to <code>x.py clean</code> to get rid of artifacts, which might not be getting cleaned up for whatever reason after toggling on parallel-compiler = true in config.toml</p>



<a name="183821986"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183821986" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183821986">(Dec 19 2019 at 07:17)</a>:</h4>
<p>(deleted)</p>



<a name="183822238"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug%20on%20highly%20parallel%20builds/near/183822238" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Bug.20on.20highly.20parallel.20builds.html#183822238">(Dec 19 2019 at 07:24)</a>:</h4>
<p>That seems to have helped, I think.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>